- name: d8.operator.prometheus
  rules:
  - alert: D8PrometheusOperatorTargetDown
    expr: max by (job) (up{job="prometheus-operator"} == 0)
    for: 30m
    labels:
      severity_level: "8"
      tier: cluster
      d8_module: operator-prometheus
      d8_component: prometheus-operator
    annotations:
      plk_markup_format: markdown
      plk_protocol_version: "1"
      plk_labels_as_annotations: "instance,pod"
      plk_ignore_labels: "job"
      plk_create_group_if_not_exists__d8_prometheus_operator_unavailable: "D8PrometheusOperatorUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      plk_grouped_by__d8_prometheus_operator_unavailable: "D8PrometheusOperatorUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      description: |-
        The `prometheus-operator` Pod is not available.

        The new `Prometheus`, `PrometheusRules`, `ServiceMonitor` settings cannot be applied in the cluster; however, all existing and configured components continue to operate correctly.
        This problem will not affect alerting or monitoring in the short term (a few days).

        The recommended course of action:
        1. Analyze the Deployment info: `kubectl -n d8-operator-prometheus describe deploy prometheus-operator`;
        2. Examine the status of the Pod and try to figure out why it is not running: `kubectl -n d8-operator-prometheus describe pod -l app=prometheus-operator`.
      summary: Prometheus is unable to scrape `prometheus-operator` metrics.

  - alert: D8PrometheusOperatorTargetAbsent
    expr: absent(up{job="prometheus-operator"}) == 1
    for: 30m
    labels:
      severity_level: "7"
      tier: cluster
      d8_module: operator-prometheus
      d8_component: prometheus-operator
    annotations:
      plk_protocol_version: "1"
      plk_markup_format: "markdown"
      plk_create_group_if_not_exists__d8_prometheus_operator_unavailable: "D8PrometheusOperatorUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      plk_grouped_by__d8_prometheus_operator_unavailable: "D8PrometheusOperatorUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      summary: There is no `prometheus-operator` target in Prometheus.
      description: |
        The new `Prometheus`, `PrometheusRules`, `ServiceMonitor` settings cannot be applied in the cluster; however, all existing and configured components continue to operate correctly.
        This problem will not affect alerting or monitoring in the short term (a few days).

        The recommended course of action is to analyze the deployment information: `kubectl -n d8-operator-prometheus describe deploy prometheus-operator`.

  - alert: D8PrometheusOperatorPodIsNotReady
    expr: min by (pod) (kube_pod_status_ready{condition="true", namespace="d8-operator-prometheus", pod=~"prometheus-operator-.*"}) != 1
    for: 30m
    labels:
      severity_level: "7"
      tier: cluster
      d8_module: operator-prometheus
      d8_component: prometheus-operator
    annotations:
      plk_protocol_version: "1"
      plk_markup_format: "markdown"
      plk_labels_as_annotations: "pod"
      plk_create_group_if_not_exists__d8_prometheus_operator_unavailable: "D8PrometheusOperatorUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      plk_grouped_by__d8_prometheus_operator_unavailable: "D8PrometheusOperatorUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      summary: The `prometheus-operator` Pod is NOT Ready.
      description: |
        The new `Prometheus`, `PrometheusRules`, `ServiceMonitor` settings cannot be applied in the cluster; however, all existing and configured components continue to operate correctly.
        This problem will not affect alerting or monitoring in the short term (a few days).

        The recommended course of action:
        1. Analyze the Deployment info: `kubectl -n d8-operator-prometheus describe deploy prometheus-operator`;
        2. Examine the status of the Pod and try to figure out why it is not running: `kubectl -n d8-operator-prometheus describe pod -l app=prometheus-operator`.

  - alert: D8PrometheusOperatorPodIsNotRunning
    expr: absent(kube_pod_status_phase{namespace="d8-operator-prometheus",phase="Running",pod=~"prometheus-operator-.*"})
    for: 30m
    labels:
      severity_level: "7"
      tier: cluster
      d8_module: operator-prometheus
      d8_component: prometheus-operator
    annotations:
      plk_protocol_version: "1"
      plk_markup_format: "markdown"
      plk_create_group_if_not_exists__d8_prometheus_operator_unavailable: "D8PrometheusOperatorUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      plk_grouped_by__d8_prometheus_operator_unavailable: "D8PrometheusOperatorUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      summary: The `prometheus-operator` Pod is NOT Running.
      description: |
        The new `Prometheus`, `PrometheusRules`, `ServiceMonitor` settings cannot be applied in the cluster; however, all existing and configured components continue to operate correctly.
        This problem will not affect alerting or monitoring in the short term (a few days).

        The recommended course of action:
        1. Analyze the Deployment info: `kubectl -n d8-operator-prometheus describe deploy prometheus-operator`;
        2. Examine the status of the Pod and try to figure out why it is not running: `kubectl -n d8-operator-prometheus describe pod -l app=prometheus-operator`.
